Audio-Visual Contrastive Learning with Temporal Self-Supervision

We propose a self-supervised learning approach for videos that learns representations of both the RGB frames and accompanying audio without human supervision. In contrast to images capture static scene appearance, also contain sound temporal dynamics. To leverage aural dimension inherent videos, our method extends self-supervision audio-visual setting integrates it with multi-modal contrastive objectives. As self-supervision, we pose playback speed direction recognition in modalities intra- inter-modal ordering tasks. Furthermore, design novel objective which usual pairs are supplemented additional sample-dependent positives negatives sampled from evolving feature space. model, apply such losses among video clips between their temporally corresponding clips. verify model extensive ablation experiments evaluate transfer action retrieval on UCF101 HMBD51, classification ESC50, robust fingerprinting VGG-Sound, state-of-the-art results.